Building the 
Open Library 



Aaron Swartz 



i build 
web applications 



1999: get.info 



2005: reddit.com 



2007: Open Library 



Open Library 



what would 
an online library 

look like? 



a wiki with 
one page per book 



quick facts 



six people (distributed) 

Internet Archive in the Presidio 

not-for-profit 

foundation + government funding 

starting with monographs 



1) infrastructure 













A BCD 
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Date ^Customer No Cases P Code 


2 


1-Oct-01 


Qaffer Inc 


_u 


75 


3 


4-Oct-01 


Fiesta Time 


19 


45 


4 


10-Oct-01 


Grogger Ent 


2 


87 


5 


16-Oct-01 


Fine Dining 


11 


99 


6 


26-Oct-01 


Fine Dining 


18 


87 


7 


31-Oct-01 


Qaffer Inc 


22 


99 


8 


1 -Nov-01 |GroggerEnt 


13 


87 


9 


U-Nov-01 iFine Dining 


12 


99 


10 


29-Nov-01 |Fine Dining 


15 


87 


11 


3-Dec-01 Qaffer Inc 


14 


45 


12 


10-Dec-01 IFine Dining 


11 


87 


13 


18-Dec-01 Fiesta Time 


13 


45 


14 


28-Dec-01 GroggerEnt 


2 


87 


15 


23-Nov-01 


Qaffer Inc 


Jr 


45 


16 


10-Dec-01 


Fiesta Time 


3 


87 


17 


11-Dec-01 


Grogger Ent 


12 


75 


18 


12-Dec-01 


Qaffer Inc 


2 


45 


19 


14-Dec-01 Fine Dining 


24 


87 
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glorified spreadsheet 



schema changes are painful 
no revision history 
type unification is tricky 
single-master scaling wall 



ThingDB 

(infobase?) 



ThingDB 





link 



user 



site 



page 



ThingDB 



oo 



book 



author 



subject 



page 



ThingDB 



/OL 1827T 

Book 



a/Ben_Fr 

Author 



./Fictiot 

Subjec 



dew /docs 

Page 



/OL 1827 

Book 









lifl 






Tom Saujuer' 












i 






a/Mark_Tu 
























subject 




/Fiction 












































folk_nouel_id F92982 




| 
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^^ 


k 
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./0L1827T 

Book 


) 


^ 
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IT^H 






iT^H 








flduentures 
of Tom 
Sawyer" 






Rduenture 
of Tom 
Saaiyer" 






Rduenture 
of Tom 
Sawyer" 






a/Mark_Twc 






a/Mark-Tw 






a/Mark_Tui 






























^^^^^^^^S 










^^^^^^| B*T^l »i B l-^i iT^T* B 




















^H 


pages 










folk_nouel_id 






aarc 

2007 
initial 


msw 

-11-05 
create 




brew 

2007- 
add /: 


/ster 

12-04 
)ages 






folkfan47 

2008-01-02 
add folk_novel_id 





everything's a thing 
schema updates are trivial 



all changes can be reverted 



time travel 



cl ass edi t: 

def GETCsel f, name): 

i f name . s tar- tsw i th C ' _ ' ) and not db . theycan t " adm in", db . cur .site): 
i-e turn requ i re 1 og i n O 

f orma t . correc tname t name , ' fsd it/') 

if name.endsiui tht ' ^add' ): 

if name == '^add': prefix = ' 

else: pi-e f i x = web . rs tr i ps ( name , " /^add " ) 

* add? 

if not db . theycan ( ' ed i t ' , db . byPage t pre fix)): re turn requ i re 1 og i n < '. 



body = ' 
reu = None 
i: 

reu = web. i nputO .gett ' r' ) 

page = db.byUersi ontname, reu) 

if page: check = page. thing 

else: check = db.cur.si 1 

i f not db. theycan t ' edi t' , check): return requirelogint 



o 



web.notfoundO 



body = 



prefix = None 



pi-int r 



def POSTCself, 



fk r\ r> 



* code.py 




scripting 



scripting 



mismatch with SQL 
specialized query language 
ORM war 
caching is difficult 



scripting 



lots of repeated code 
objects 

map URLs to objects 
1 locate the objects 
display the objects 
edit the objects 



scripting 



lots of repeated code 
accounts 



create user accounts 



log in users 
forgot password? 



cl ass edi t: 

def GETCsel f, name): 

i f name . s tar- tsw i th C ' _ ' ) and not db . theycan t " adm in", db . cur .site): 
i-e turn requ i re 1 og i n O 

f orma t . correc tname t name , ' fsd it/') 

if name.endsiui tht ' ^add' ): 

if name == '^add': prefix = ' 

else: pi-e f i x = web . rs tr i ps ( name , " /^add " ) 

* add? 

if not db . theycan ( ' ed i t ' , db . byPage t pre fix)): re turn requ i re 1 og i n < '. 



body = ' 
reu = None 
i: 

reu = web. i nputO .gett ' r' ) 

page = db.byUersi ontname, reu) 

if page: check = page. thing 

else: check = db.cur.si 1 

i f not db. theycan t ' edi t' , check): return requirelogint 



o 



web.notfoundO 



body = 



prefix = None 



pi-int r 



def POSTCself, 



* open 

book = tdb.Thing.get(<b/0L1827T') 



ui ew 



print '<a href="$book .url ">$book . t i tle</a>' 



* edit 

book. pages = 275 

book . saue C ' add pages ' ) 



* query 

1 ong_books = tdb . Th i ngs C 

tdb.k. pages > 200, 

type= 'book ' ) 



caching for free 



store items, queries in RAM 



on save 



invalidate item cache 
invalidate matching queries 
broadcast invalidations 











your app 






infogami 




ThingDB 









scaffolding for free 



URL mapping: URL == name 
view/edit: types 

1 schemas -► default templates 
custom templates 
1 user templates 
history/diff: free 



accounts for free 



infogami handles it all 

your app code exists in plugins 

lots of hooks 



$for p in page .type .proper t i es: 
$ key = p. name 
<tbody> 

$if not p .get( 'un ique ' ) : 
$ i ="0" 
<tr> 

<td>$key</td> 
<tdX/td> 
<tdX/td> 
</tr> 

$for x in page. d [key]: 
<tr repeat="$ i " > 

|<td colspan="2" styl e="paddi ng-1 ef t: 1.0em"> 
$: thing input (p .d . tqpe, key + "*" + strCi), ualue=x></td> 
■ ■■ITTi+TT^TTTT ^TTTTIIHHf ? J TTTTT ? 

<td> 

<button type="remoue" tempi ate="row^$key">Delete</button> 
<button type="moue-up" tempi ate="row^$key" oncl ick="retur 
moueup.apply^ th is, [euent 1 > ; " >Moue Up</button> 

<button type="moue-down" tempi ate="row^$key" 
oncl ick=" re turn mouedown .appl y( th i s, [euent ]>; "Xloue Down</button> 
</td> 
</tr> 
$ i = i + 1 



HTML templates 



HTML templates 



skinning is difficult 

web designers can't touch SQL 

web designers need SVN 



put the templates 

in the wiki 



templates in wiki 



web designers can work away 

skins come for free 

users can create skins 

tied to data, not code (time travel!) 



publishers: ONIX 



HaperCollins 

Simon and Schuster 

Random House 

Cambridge University Press 

Penguin 

Wiley 

Thomas Nelson 

O'Reilly 

Elseveier 



ibraries: MARC 



Library of Congress 

Montana State 

UNC 

Drexel 

University of Toronto 

MBL/WHOI 



ibraries: MARC 



Talis! 



book reviewers 



New York Times 

Harper's 

New York Review of Books 

Reader's Catalog 



ftOO http: //ia3 313 IB . u s,archjve."0rg /fli pbook/ fl ip book-p h p^identi Fier . . , s/lovef orho u ris loveQ Gbarri aJa&d atas e nrer-ia33 1 3 L8. us.archive.ofig 
j_+J [_+J | C* I r hltp:/yia33il3 ia.us.archive.o r g/Flipb[Kjk./flipbDok,php?icenE ^ier = lovieforhourislove(>0barrialajdidatafialh = /2, 1 ' ^ 0,*Google 




LOVE FOR AN IIOL'R IS LOVE 
FOREVER 



CHAPTER I 
"who eider hv with royai. jukT" 
" Wk» (Uh by wU i»tJ ifcf- 



' Tbcnfjb (■»* ■*! trawrn^ mad Jeiifc. miy 
Let* br ib bow fa tow fmw.' 



BETWEEN the leave* of an old romance I fouml 
one day the shadow of a tily and a song. The My 
grew forty year* ago, the aang *u song a* it was | 
ered, The fewtr it nearly daft, the medf have nearly 
faded away, hut the sJory they keep u unfofgotlen. 
For in becoming "" Life " it made kstti eternal. 

Befof e [he flower Idoumcd, before the soflg had found 
b voice, Francesc* Alhettnn had diTamed utf love, m 
saints dream of heaven — wonderful, mystical, Far off — 
on object both of fervent dam and of wiMful fear and 
uncertain! f. Tt$ her young li» had hern peopled frwm 
noble book*, and il was in then - pages she had. mer I «r 
friends and companion* — men, romantically. honor.iMr 
and U lyal , women, fail hi ul En tote, even unto death — ■ 



J 






Maq" Fy 



Liitan 



3) processing 



parsing 



lots of Python scripts 

convert to intermediate dictionaries 



merging 



MELVYL algorithm 
impressive success 
dupes even in LC 



FRBRizing 



coming soon... 



4) display 



I C | http:iydema.openlibrarY.Drgj 1 



Open Lfhwry 



What's New: 4 r- inutei &$b - b/P*wer ivHtn ftvaluatrtn (Vrevvw/l upd&led by G dm; 



Open Library (Open Library) 



I Q,»CrDogle 



^i K iVm site is an early technology preview , and many things are still influx or not operational. In the meantime, you may 

\ ) t' ' want to take a guided tour. Please check back as the site de uelops. Thank jyo u. r 






The 
OPEN 



Lib 



ra 



y 



Open Library 



Imagine a library that collected all the world's information 
about sill the world's books and made it available for everyone 
to view and update. We're building that library. (Mure...) 



against l©ve| 

□ FuHtext items only ► advanced search 

{Or take ;i guided tour of the site...) 



[ 



(.(I 



~ Recent AVwj ~ 



New features! 

In prr>pn-n»bLnn far the OCA 
nettfu un October 17. we'vt 
updated tlic site to include lots 
of juicy new teature-s. (Rend 
more-....) 



r^55^ 



Mild Books Build The Library Develop The Site 

t Recent chang es t About the project ► Bug tracking 

► Example search * About the technology l» Source code repository infcgami: 

► Guided tour * About the librari airship ► Documentation 



Library 



AboatLrs lIAIOCAICCIWMFl 



I + J I C | hit p:/j'demo.open!ibrarY. orgy b/Agiinst_LDve_0 



Open L&wry 



Against Love (Open L i b re. > : 



I Q,» Google 



nttfy] \h& nociflefi MAy 4 



i ,i i 



p«^° ; 



rJmstfe is aneaWi/ fecAiwcOjjy jsretrt'euj, and many fjhtnjjs are tftiff injtuxornol operational. In the meantime; you 
may want to take jqu ided to ur. Please check back as the site develops. Thank you! 





fl 


ro 


1 


AflAIW|T 
LOVE 


p 






Iaupi Kifnfi 


^ 


* 


'4 


•'. 



Against Love 

by Laura Kipnis 

Available Metadata: t: i ht 

Number of Pages: 224 
BISACi ['S0COO2O10") 
Publisher: Knopf Publishing OroLip 

Publish Date: eoq.iqBzG 
Number of pai^cs: 2.2A 

1SRN 1 0:03754 2 iH^O 
CTLN 14:09780375421691 
liJUIN 13: 978037542 ia^i 



" Option* 



Ml 1 

Am n jjin 
Abt Books 
Barnes ir.d -Sebic 
Alihrix 



Pencils 



llUHHUVt 



Yoor local, lib-racy 

BoofcMoocb 

J'i:! t -Vr.i:Lir 

DOWNLOAD 

rial eivuilublefur this bixik 



KKV1E1T HOOK 



Laura Kipnis Is a professor of media studies al Northwtslcrii University. She has received fellowships and grants from Ihc 
(Siiggfinhciini Hounds! inn, the ttncikflfcltar Etoimctatinn, and ihc National Imdowmcnt far the Arts. She has published numerous, assays 
and articles on sexual politics and contemporary culture both here and abroad. Chapter One 

Love's Labors 



• 













™* Options *~ 

HUT 

Amazon 

Abe Books 
Barnes and Noble 
Alibris 

Powells 

HORROW 

Yout local library 
BookMooch 
Title Trader 

DOWNLOAD 

not available for this book 
















Laura Kipnis (Open Library) 



hit p: / y de rno.a pe nl i b rary. orgy ay Lau ra_Ki p ni s 



*■ Q. T laurakipnii 



HiA^y] laii t-Kicifiet. 10 stands Ago 



i ,i i 



pe** ; 



Sit* 



Zftisstre is ansarty technology preniewi, and many things are slitl injluxor nal operational. In the meantime, you 
may want to take aou ided to ur. Please, dieck back as the site deuelaps. Thank you! 




Laura Kipnis 



i: v it 



Website: 

|]Uu://w^ , w.cQniinunLcatLC] n Jioith wt'stcrn.^dW rtf/ faculty/Laura Kipnis/ 



Location: Evanslon, EL 



Alternate Name: 



~ Options 



CicnreH: 

Am yet available. 

KchilrdAuLhur*: 

Not yet avaiiabte. 



'&w 



Beaks By This Author ~ 

Against Love by Laura Kipnis 



Against Love 



(Knopf Publishing Group, 20040914) 



J! 



Against Love by Laura Kfpnfa 

(Knopf Publishing Group, 20030826) 

The Female Thing by Laura Kipnis 
(Knopf Publishing Group, §006101?) 



• 



eon 


Browse by Subject (Open Library) 






| C | Q http:./;inviE, ble.netyopenlibrary/iubj 


ect-brnwse ,' drn Vrie . ht ml 


HQ,v Google 


_ o 


Open L$r&ry 









[*air.][h .nary! last «iaaifi«a Alki . 




1 


iitj 


Browse by Subject- Please select 


a topic of interest. 






► General Works 


► Music 






► Philosophy, Psychology, and Religion 


► Unci Arts 






Philosophy (General] 


► Laiiftuajy: arid Literature 






Logk 


► Science 






Speculative philosophy 


► Medicine 






Psychology 


► Agricti]ti]r£i 






Aesl holies 

T~i l_ : , - 


V Technology 






ht rues 

Religions - Mythology - Rationalism 

Judaism 


► Military Science 

► Naval Science 






Islam - Bahaisii] - The ositphy. etc 


► Bibliography, library Science, and General Information 




*■ 


Buddhism 


Resources 






Christianity 








The Bible 








Doctrinal Theology 








Practical Theology 








Christian Denominations 








t Auxiliary Scicrax-s of 1 listory 








* General and Old World History 






* 


> Hinlnrv nt \mfiTim 






V 





Open Library - FRBRization - Relationship Link Map based on Faceted Classification 



daMbaf ! return towiki viewl 



dashboard dewing link map- return 



D 



i , The adventure! of Tom Sawyer / ill ustrations by Paul Geiger r afterword by Bernard DeVoto, 
by Mark Twain; Paul Geiger 

Language: English Type: BookiFittionJuTenileaiidierice 
Publisher; Pleasantwlle, N.Y. : Reader's Digest Association, (S 1 985. 



Audio 






D 


D 



D-D 



Internet Resource 



D 

D 

D 



Br. 







□d 

n 



D 

D 

D 



Software 



D 

D 



Related Titles 




nboo-..- 


D Book Title rote 




□ □□ 


Q Sopk Title note 




ODD 


□ Book Title note 




DOB 


□ Book Title note 




□ E3D 


□ Back Title note 




□ 



D Qn D 

Books 

0n D D 
□□ D n D 

D I' 



Legend - select facet to view relationships 

□ Show full Text 

• Show Same Authors 

□ Show Same Subject* 

□ Show Same Publishers 

□ 5how 5imilar Date Range 

□ Show Same Genre 

□ Show Nearby Books (Geotag) 

□ Break Relationship Link 

□ Add New Relationship 

□ Compare Relationship Maps 

□ Bookmark This Map 



overlay your tag s - overlay tags from friends - overlay al I tag s 



Edit b/A9iinst_Love_0 (Open Library) 



BOO 

I + J I C | hit pi/ydemo-openlibrarY. orgy b/Agiinst_LDve_0'm='Edil 



Open L&wry 



I Q,» Google 



Htfy] liil nociflefi MAy 4 



i ,i i 



pet* ; 



SiU' 



rJmstfe is aneaWiy teoft]Dor.r>jjy jjref seiu, and many things are stfff trt/Jiu: or nol operattona/. /rt the meantime, you 
may want to take agu ided to ur. Please check back as the site deueiap-s. Tha nk yo u. r 



Varc type type/edttiQD 



1C 



Title Against Love 



Subject 

Scan Contributor 

DO/ 



Source Name r:indcm]linu^^rj07f»i]_]fl2fk)9-t ; l."]JjC'A']'-bXriJ.-2i 



r 



G7YJV 14 Q»7gQa.754aiBgi 

JVon Sponsor 

EtHiian 

Language Code 

Source Record 

Work 



Conlri&utians 

OCA identifier 
LCCN 
Physical Extial 



Publish er Kiicpf iVbuahlnp; Group 



del Ere 



Add subjeti 



[IKI.KTK 



Add ciintribuLLuns 



Laitr< &snte is a professor of media shtdict at Kgrthwcsteni Urovcreifr. She hi" •gemyied jj 



* 



BOO 



n Library 



type /edition (Open Library) 



http: / /de m o .0 pen iibraiy . org / cv p e , r c d iiion 



[Q^ Google 



L 



[edit ;n story; ait TiodiTied Nov*mc*' J r 2007 



i.ik 



mK 



This site, is on eonfy tevhnnlagy preview, nnd many things are still influx nr not npervfionnl. In the meantime, 
you may want to take a guided tour. Please checkbackas the site develops. Thank yaut 



Currently viewing the page: 

type/edition 



Name "* 


Typt « 


description 
is_primitivc 


properties 
properties 


title of type type/strine; 
subjcct[ ] of type type/ string 


properties 


scan_contribLitor at type type/ string 


properties 


DOI of type t> pe/s4ri ng 



properties 
;o:;:i'rl:.- 
properties 
|]rnpt;rtie-Ji 
properties 
properties 
properties 
properties 
properties 



source name of type type/string 
GTIN_i4 cdype t>p*/sfri n^ 
scan sponsor of type type/string 
edi turn if type- ly[*/ rfl rlii£ 
Language code of type typc/string 
sou rce_recof d of type- type/ string 
work of type type/work 
contribution^ ] of type type/string 
oca identifier of type type/ string 






Open Library 



edit Edition View Template (Open Library) 



h[i p: //demo-.ope nl i brary.o rg /type; a tf i t i on/v ie w .t m pP m = edi t 



Ol--Cchfl^le 



Welcome, Aaron flwarti I Frafe. 



Yl-o-w [history] [modify template' 



■: . i f 



L 



p 



err* 






Tftts site is an early technology preview. Any changes that you make will take effect, but could possibly 
be lost at some point as we continue to develop the site. Thank you for your pctt iencSr 



Currently editing the template for: 

Edition View Template 

Fafie type hp<!/tempLate ( Change ) 



Title Fdirion View Template 



$d«f with [pa^e J 
$y&F titter S-pagt-title 

^^^_^]ei*leetf7altie/cs5i/re»lfclJ(^ , ] 

■ca name- "top" >^/a> 

<div id-"book-art"> 

idiv id -"boak'S 

Sir fwge.cl.fcetC' IS RN F _io"); <a £M*f=7«atic^})fiijla^ Sps^e JSBN_ioI iJ/S{pinge.[SB?J_io>-jp* "> <j 

^=75tatic/hcat^ere/fuU^^ ah- "Spase .title' border- V 

height- "27.'5"^ida3-"i85" />■:/&:■ 

</div> 

<br cteaT-'all" / > 

■c div id - "vicivthisbcoktiLn" > 

<a hrejf-'Vaddr^iew?edibfln-$p^e.name , ><.in!i£ anc-'7static/imaftes/r<!^icsvbod-Lpng' alt- 'Kevi ew Book* border -"0" 



flifl 



J 



ftOD Tne soul of rnan under socialism (Open Library) 

| C | L O http: /yde mo.openlibrary.Drgt:aO 80/ stalic^stodV Bcan_book_authenticale.html 



Open Lfiirffry 



kstorv] Ian modified July 17 



I Q,» Google 



" 



i ,i i 



D 



tin" fc 



Zftisstre is aneaHy teohnfftojjy jsref i'^ u.> . i4ny eftonycs (ftm you make uriW take effect, but could possibly be lost at some 
point as we continue to develop the site. Inank you for your patience. 



The 

OPEN 

Library 



Thank you for your 
patronage, brewster. 




UNC 



SniOIII 4M IM MKM.il ION 
,VM> I IHKMtY S[IKN<K 



Please review the following information before proceeding: 

Scanning-on-demand system for the Open Library is done in partnership with the 
University of North Carolina School of Information and Library Science. This book is 
available from the University of Toronto Libraries. 



You have offered to sponsor: 

The soul of man under socialism, The socialist 
ideal— art, and The coming solidarity fbv wiide. 

Oscar. 1 

The scanning time And cost estimate is: 

(48pp. x 10$) + $5 = $9,80 



* 



bggS*-\£M 




Z-> *z ;• ^ 








il 



v>>* 






SA 



HELP WANTED 



we want your data 



we want your time 



we want you 



demo.openlibrary.org 



infogami.org 



demo.openlibrary.org 



me(a)aaronsw.com 



